nit: it's minus 8 bytes for the first "entry", not "slot", right?
That should be VersionEntry, if I'm not wrong. Should we add something like SERIALIZED_SIZE to it (like we have for Header)?

starting_slot is 8 bytes,

This has nothing to do with an VersionEntry entry to my understanding

Reading it again I am so confused, but I know it is 8 bytes

this logic doesn't seem right for me. Maybe I'm missing something, so if you could clarify:

1. with current logic, first slot will always be occupied (because check at line 124 will always be false). I don't think that's correct.
2. are we sure that indices[0] can't be lower than 8 (in which case we would panic)?

with current logic, first slot will always be occupied (because check at line 124 will always be false). I don't think that's correct.

The first index isn't an offset of 0, it is an offset of 8 because starting_slot

are we sure that indices[0] can't be lower than 8 (in which case we would panic)?

Yes because it will always be 8 bytes higher than starting_slot

I looked more into this, and after debugging bytes and looking for multiple example, I think I figured it out.

First, the indices are negative integers and we should update indices field in both SlotIndexBlock and SlotIndexState to [i64; X]. It even says so in the spec:

A SlotIndex record may appear in a stand-alone file which by convention ends with .e2i - in this case, the offset is counted as if the index was appened to its corresponding data file - offsets are thus negative and counted from the end of the data file. In particular, if the index is simply appended to the data file, it does not change in contents.

Second, if index (offset) point to the beginning of file (meaning index == position_of_slot_index_block), then that block is skipped.
In most cases, first block is present, and will be 8 bytes from the start of file (VersionEntry takes first 8 bytes), resulting in your math working but with the wrong reasoning (it's not because of the starting_slot but because of the VersionEntry).
However, if first block is skipped (if I'm not mistaken, epoch 819 has starting slot 6701056 which is empty), this will not work, because first block already points to the beginning of file and we don't need to subtract 8 bytes.

Now, the solution. For this function to work, we need to know at what byte the SlotIndexBlock start in the original file, and we have 2 ways of doing so:

1. sum encoded sizes of all entries up to this point (version + blocks + state)
2. know length of the file and subtract encoded sizes of SlotIndexBlock and SlotIndexState entries (which are fixed: 8192*8 + 3*8 = 65560 and 4*8=32)

I think it's a bit hard to explain this in words, but I tried my best. If something is not clear, we can have a call and discuss it.

I choose the 2nd option

All of these should be constants in various places. For example:

• SlotIndexBlock should have something like:

  SERIALIZED_SIZE: usize = 8 * (1 + SLOTS_PER_HISTORICAL_ROOT + 1);

  which will be used in impl TryFrom<&Entry> for SlotIndexBlockEntry

• SlotIndexBlockEntry should also have

  SERIALIZED_SIZE: usize = Header::SERIALIZED_SIZE + SlotIndexBlock::SERIALIZED_SIZE;

Similar for SlotIndexStateEntry.

The indices/offsets should really be i64. Combined with comment above, this would be:

let beginning_of_index_record = file_length - SlotIndexBlockEntry::SERIALIZED_SIZE - SlotIndexStateEntry::SERIALIZED_SIZE;
let beginning_of_file_offset = -(beginning_of_index_record as i64);

And later you would just check if *offset != beginning_of_file_offset {

nit: instead of collecting all of these into Vec<u64>, than converting each into [u8;8] and than each of those into Vec<u8>, and finally collecting all of them into one Vec<u8>, I think we can do something like this:

let mut buf = vec![];

buf.extend_from_slice(&self.slot_index.starting_slot.to_le_bytes());
for index in &self.slot_index.indices {
    buf.extend_from_slice(&index.to_le_bytes());
}
buf.extend_from_slice(&self.slot_index.count.to_le_bytes());

Ok(Entry::new(0x3269, buf))

Considering that there is likely no big gain in performance, and my suggestion might be a bit less readable, it's up to you.

Also below.

I think this is better 👍